Determining Assessments

Now that the learning objectives have been developed, our group should now determine the best assessment instrument(s) for our curriculum design proposal.

Consider the principles of effective assessment design described in Chapter 6 of Instructional Design, as well as in the article, “Seven Practices for Effective Learning.” Think about the characteristics of effective assessment instruments and the different formats available. Based on your knowledge, which format(s) do you think is/are best suited for your group’s curriculum? What type of items do you think should be included within these formats to assess learners’ achievement of the learning objectives?



Assessments To Support Learning Goals and Objectives (final draft 9-26/ Edited by LC 10-1)

Volunteers will be able to operate museum computer systems and manage an Excel database of patron information.


Assessments: · Diagnostic Test to determine the learner’s level of computer knowledge prior to taking the class. Computerized for assessment of technical skills (typing, email, and computer navigation)
Sample instrument questions: Can the learner navigate to the email program on the computer? Did the learner use spell check before sending email? Did the learner save the Excel document after entering the data?

· Performance Assessment – Using example data provided, learner will enter data into excel database which will be reviewed by instructor for accuracy. (Rationale: learner can follow procedures to insure accurate entries of donations and maintain patron information)
Assessment content examples would include observing the learner enter data into correct fields and learner reviewing for accuracy.

1. Learner will respond to 100% of patron emails within 24 hours of receipt.
· Learner will demonstrate the ability to type at a minimum of 30 wpm with 95% accuracy.
· When provided with a jobe aid (example text), learner will respond in a professional manner every time.
2. When provided with a job aid (example entries), learner will demonstrate proficiency in recording patron donations with 100% accuracy in the appropriate museum computer Excel database program.
Volunteers will lead tours of museum exhibits and respond to patron questions.
Assessments: · Pencil & Paper Assessment – Learner will complete a 20 question quiz requiring them to match specific artists with their paintings and exhibits. (Rationale: volunteers will demonstrate familiarity with museum exhibits)
Sample: quiz would have 20 pictures of artwork and 20 artists' names. Learner will be able to match artist to artwork (fill in the blank next to artwork from list of artists' names).
· Performance Assessment – Learner will work in groups to conduct tours of the museum. Instructor will evaluate their performance using an observation rubric and checklist of items learner are expected to perform. (Rationale: volunteer will demonstrate their ability to interact with the people and guide the visitors systematically through the museum exhibits, as well as how they adapt to the groups' dynamics)
Sample checklist questions: Did they greet the patron courteously and introduce themselves? Did they answer the visitor’s question correctly during the tour? Did the correctly identify the artwork in the exhibits?

1. Given a printed list of descriptions, learner can match artists to their paintings in the exhibits with 70% accuracy
2. Utilizing a printed job aid (map) learner can summarize museum layout and museum collections to visitors 9 out of 10 times.
3. When approached by a visitor, learner can greet the patron and provide the information so that the visitor walks away satisfied 9 out o f10 times.
· Learner will recall information to answer common questions
· Learner will identify the appropriate resource available on the museum computer or job aids to answer the patron's question 9 out of 10 times
Volunteers will work in telephone and email campaigns to raise funds from museum patrons.
Assessments: · Performance Assessment - Learner will demonstrate their communication skills in summarizing the museums's fund-raising needs and responding to objections. Learner will input patron donations into Excel database to determine level of accuracy in maintaining the computer database. Instructor will observe performance and complete a checklist of required items to record learner's accuracy.Sample instrument questions: Donations will be used for what? The museum's current fund-raising goal is $xxx? The same observation checklist from Goal 1. could potentially be used here - using the relevant Excel items. Assessment content examples would include observing the learner interact with another group member using a script to begin with in relating the fund-raising needs. When entering donations, learner would be observed entering data into correct fields and learner reviewing entries for accuracy.
1. When provided with a script, learner will be able to summarize the needs of current museum fund raising programs to the patron 100% of the time
2. When provided with a sample script, learner will be able to respond to objections 7 out of 10 times.
3. When provided with a job aid, learner will demonstrate proficiency in recording patron donations with 100% accuracy in the appropriate museum computer database.






Scenario:


The directors of an art museum, specializing in 19th and 20th century American painting, would like to institute a new, formal training program for their volunteers. The museum directors are planning to hire a curriculum design company to create this training program. The museum directors would like volunteers trained to perform the following tasks:

· Leading tours of exhibits
· Staffing the museum “information station” to answer visitors’ questions
· Maintaining an Excel database of patron information
· Calling and e-mailing patrons to raise funds


What would be the most effective way to design curriculum for this program? You and your group members, as “employees” of a curriculum design company, will propose your curriculum design ideas to the museum directors in a PowerPoint presentation.

By Monday, the Week 3 Facilitator should create a page in the group wiki designated for “Assessments.”

By Wednesday, post to your group wiki a description of 1–3 assessment instruments that you think should be included in your group’s curriculum design proposal. For each instrument, identify its format, describe the general content it would include, and incorporate at least one sample item. Include a brief rationale explaining why you believe each of your suggestions would be effective.
Return to your group wiki and review the assessment suggestions posted by your group members. How do your ideas compare with those of your colleagues? Do some ideas overlap? Do the ideas of your colleagues cause you to have a different perspective?

By Friday, each group member should post his or her suggestion for the group’s final assessment instruments (the group must present at least two in the proposal). These suggestions should reflect a combination of what you consider to be the best ideas from all group members.
It is the responsibility of the Week 3 Facilitator to review each group member’s suggestion and create a final set of assessment instruments that reflects the majority opinion.

By Sunday, the Week 3 Facilitator should post the final list of assessment instruments to the group wiki. Each assessment instrument should be accompanied by a description of its format and general content, at least one example of an item that would be included within it, and a brief rationale for why it would be effective. Group members should visit the wiki, review the list, and use the wiki or other communication means to resolve any disagreements. In addition, the Facilitator should post the URL of this wiki content to the Week 3 area of the Group Project discussion board.


Week 3 Curriculum Design Proposal Task Summary

Monday
Tuesday
Wednesday
Thursday
Friday
Saturday
Sunday
Week 3 Facilitator only
Create an “Assessments” page in wiki
COMPLETED





Review group members’ suggestions for final set of assessment instruments. Create and post a final set of assessment instruments that reflects majority opinion. Post URL of this wiki content to Week 3 area of the Group Project discussion board.
All group members


Post a description of 1–3 assessment instruments for group’s curriculum design proposal.

Review group members’ assessments postings. Post suggestion for group’s final set of assessment instruments.

Visit the wiki and review the Facilitator’s posting of the final set of assessment instruments. Use the wiki or other means of communication to resolve any disagreements.


Group Ideas for Assessment Instruments
Name
Assessment Instruments
Taresa Breedlove
1. I think there should be a group activity completed for the tour portion. I would like to see each volunteer take turns with a small group of other volunteers doing a mock tour so they could practice their skills. This would help confirm that they know how to use their information help guides and how to tour people through museum exhibits.
LC I agree, there should be group activity/on-the-job observation. I would suggest that they should first participate in tours with experienced guides - (but I suppose that part isn't part of the assessments : )
AK: Looks like we’re all in agreement on this one.
2. I would suggest a test or quiz on the excel database portion. I think the volunteers could complete a test where they input information into an excel spreadsheet to show they know how to set it up with patron information.
LC I agree a paper-and-pencil test (computer) would be a good fit here.
AK: Pencil-and-paper assessment for the Excel portion, but does this address the complete objective of “operates and manages computer systems?”
3. I think a question and answer session might work for the "answering patron questions". The instructor could create some question cards and the volunteers could even work together in groups to practice answering questions.
LC I think this could work well - and the group setting could help provide ideas that may not be first nature to everyone. Learning from each other is a good potential benefit.
Lori Czuba
1. Preassessments would be a good tool for volunteers doing computer tasks. Preassessments could measure typing speed and accuracy, and knowledge of spreadsheet/database programs for recording donations. The results of the preassessments would be used to determine what specific training the volunteers would need to complete.
JR An Entry Skill Test can be given before or along with a pretest assessment prior to any lessons.Entry level skills can be assessed in an application process An example - applications are done on line only designed to cover some basic skills like typing, emailing, ect. Also offering a list of volunteer job responsibilities in advance.Pretest done after job placement has taken place
AK: Definite entry skills and preassessment for everyone seeking a volunteer position with the museum.

2. Leading tours, working in the Information Station, and working on the fund raising programs would lend themselves to Performance Assessments in the way of simulations (role playing or scenarios). Feedback throughout this process would help to equip the volunteers to provide the best experience for the museum visitors and patrons.
AK: I think this assessment is very closely matched to my suggestion below.

3. For the objective of matching the artists to their paintings, a simple pencil-and-paper assessment would be a good measure to know the volunteers are ready for the visitors.
JR -Pencil and paper assessment ( computerized) :A matching exercise - matching artist facts with paintings ( Recognition Items)Fill-in-the-blanks - Information about the museum ( Recall)Scenarios - written responses to likely museum situations ( Constructed Answers)
AK: Agreed—pencil-and-paper assessment.
Annette Kristofer
All three objectives required criterion-referenced assessment instruments, because we are trying to determine competency in a skill or knowledge, not ranking among learners. The three objectives would require a combination of performance and pencil-and-paper tests.All volunteers should be given an entry skills assessment to be sure they have the requisite skills. This could be administered through an interview process using a checklist. Preassessments can then be used for the computer training to determine the existing knowledge volunteers have of computer components, typing ability, and data entry skills.Objective 1:
Volunteers will be able to effectively operate and manage museum computer systems. This would be assessed through observation of on-the-job performance. This assessment puts the volunteer in the “real world” to use the skills they were instructed on to complete the tasks. A checklist would be used by the instructor to evaluate the volunteer’s actions. Sample: Did they login with the correct password? Did they enter the data into the correct field? Did they save the file after entering new data? Did they spell-check the email before sending? Did they address the patron professionally in the email?LC This could be on-the-job, but what if we use some previous (perhaps difficult) emails and patron donations in the non-live environments to get a feel of their style, professionalism (for the emails) and attention to detail on the excel work? In this way, we could identify where they may need more coaching in communicating or if they need additional training in excel, etc.
AK: All “volunteers” would take an entry skills assessment to determine in which role they might best fit at the museum.

Objective 2: Volunteers will effectively lead tours of museum exhibits and answer patron questions.
This objective requires two assessments: a paper-and-pencil (or computer-based) assessment combining recall and recognition items to identify the correct answer from an alternative and/or recall the information presented during instruction. The second part would be a simulation assessment through role-playing to conduct an assessment of the learner’s ability to lead a tour. A checklist would again serve as the means to verify the knowledge acquired. Sample: Did they greet the patron courteously and introduce themselves? Did they answer the visitor’s question correctly during the tour? Did the correctly identify the artwork in the exhibits?LC I can see a paper-and-pencil assessment for matching the artwork to the artists, but I'm not confident about the tours and visitors questions. I really want to see how they interact and respond to people in these situations, rather than what answer they can write down.I definitely agree in role- playing for any of the visitor interaction positions.
AK: I think the role-playing that I suggested for the second part of the assessment would be the interaction that you are looking for—other learners would each ask questions while the “tour leader” would answer the Qs—providing interaction. The instructor would have a checklist to confirm correct response/action or provide feedback where additional training is required.

Objective 3: Volunteers will be able to effectively work in raising funds from museum patrons.
The paper-and-pencil test (recall items) would assess the learner’s ability to summarize the museum’s fund-raising needs and responses to objections through short-answer or fill-in-the-blank questions, such as: The museum’s current fund-raising goal is _? How will the money contributed be used? I don’t contribute to fund-raising campaigns because I don’t support the arts. The same on-the-job assessment checklist used for the first objective could be used here for assessment of the learner’s computer proficiency. LC Seems reasonable for paper-and-pencil with recall items here.

Rationale: The checklist would verify that the acquired skills were “observable” when completing the terminal objective. Continuous feedback could be provided during this time to guide the learner as necessary. The paper-and-pencil tests would be “graded” through number of correct answers versus number of incorrect answers.
Janice Rhodes
1.Diagnostic Assessment ( pre assessment) for Data base maintaining and emailing tour information.
Rationale: Diagnostic testing will help designers plan and guide instruction by obtaining learners learning preferences, prior knowledge and skills, and any misconceptions that need to be addressed during instruction.)
The format can be Pencil and Paper ( computerized):
True/False - for misconceptions about the museum and the job
Computerized - for motor skill assessment ( typing, emailing, computer navigations)
Completions - to construct written responses to simulated questions about job responsibilities
This can cover determined entry level skills, like typing and emailing.Maintaining the Data base can be covered during training if it is determined that more need it than not.

2. On the job Observation: Rationale: This can be done on the mock tours. Using a rubric and checklist with a rating scale to assess the quality of volunteer interactions with people.
3.

Name
Final Assessment Instrument Suggestions
Taresa Breedlove
1. For leading the tours - I like a performance assessment. I really think the volunteers should do some mock tours, or some type of activity that shows they have the ability to lead people through the museum exhibits. The specifics could be that they break out into groups of three and each gets to be a tour guide. While the tour is taking place, one volunteer would be an observer and would complete a checklist and then provide feedback to the guide at the end of the session.
2. Staffing the station and answering questions. I'd like to see maybe two pieces to this. First a quick review to prep them for answering questions. Then they could complete a pencil/paper test on the information.
3. Maintaing the excel database. I agree with the suggestions of completing a test where they fill in answers into the database, etc. This would show they have the ability to work with, and maintain the database.
4. Calling and emailing patrons to raise funds. I also like the pencil/paper test on this. I'm not sure about the emailing part though. Do you think it could be feasible for them to do another performance assessment sending sample emails to show they know how to do this task properly? I just think it would be important to ensure we have lots of performance-based assessments when students are required to "perform or demonstrate" a task.
Lori Czuba

Annette Kristofer

Janice Rhodes
1. Leading tours of exhibits - Performance assessment.
2. Maintaining an Excel database of patron information - Diagnostic Assessment including Pencil and Paper ( computerized), True/False, and Completions